Method, apparatus, and manufacture for smiling face detection

ABSTRACT

A method, apparatus, and manufacture for smiling face detection is provided. For each frame, a list of new smiling faces for the frame is generated by performing smiling face detection employing an object classifier that trained is to distinguish between smiling faces and all objects in the frame that are not smiling faces. For the first frame, the list of new smiling faces is employed as an input smiling face list for the next frame. For each frame after the first frame, a list of tracked smiles for the frame is generated by tracking smiling faces in the frame from the input smiling list for the frame. Further, a list of new smiling faces is generated for the next frame by combining the list of new smiling faces for the frame with the list of tracked smiles for the frame.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/423,039, filed Mar. 16, 2012, titled “METHOD, APPARATUS, ANDMANUFACTURE FOR SMILING FACE DETECTION”, now U.S. Pat. No. ______, whichis incorporated by reference herein in its entirety.

TECHNICAL FIELD

The invention is related to object recognition and tracking, and inparticular, but not exclusively, to a method, apparatus, and manufacturefor smiling face detection in which the object classifier is trained todistinguish between smiling faces and all objects in the frame that arenot smiling faces.

BACKGROUND

Electronic cameras image scenes onto a two-dimensional sensor such as acharge-coupled-device (CCD), a complementary metal-on-silicon (CMOS)device or other type of light sensor. These devices include a largenumber of photo-detectors (typically two, three, four or more million)arranged across a small two dimensional surface that individuallygenerate a signal proportional to the intensity of light or otheroptical radiation (including infrared and ultra-violet regions of thespectrum adjacent the visible light wavelengths) striking the element.These elements, forming pixels of an image, are typically scanned in araster pattern to generate a serial stream of data representative of theintensity of radiation striking one sensor element after another as theyare scanned. Color data are most commonly obtained by usingphoto-detectors that are sensitive to each of distinct color components(such as red, green and blue), alternately distributed across thesensor.

A popular form of such an electronic camera is a small hand-held digitalcamera that records data of a large number of picture frames either asstill photograph “snapshots” or as sequences of frames forming a movingpicture. A significant amount of image processing is typically performedon the data of each frame within the camera before storing on aremovable non-volatile memory such as a magnetic tape cartridge, a flashmemory card, a recordable optical disk or a hard magnetic disk drive.The processed data are typically displayed as a reduced resolution imageon a liquid crystal display (LCD) device on the outside of the camera.The processed data are also typically compressed before storage in thenon-volatile memory in order to reduce the amount of storage capacitythat is taken by the data for each picture frame.

The data acquired by the image sensor are typically processed tocompensate for imperfections of the camera and to generally improve thequality of the image obtainable from the data. The correction for anydefective pixel photodetector elements of the sensor is one processingfunction. Another is white balance correction wherein the relativemagnitudes of different pixels of the primary colors are set torepresent white. This processing also includes de-mosaicing theindividual pixel data to superimpose data from spatially separatemonochromatic pixel detectors of the sensor to render superimposedmulti-colored pixels in the image data. This de-mosaicing then makes itdesirable to process the data to enhance and smooth edges of the image.Compensation of the image data for noise and variations of the cameraoptical system across the image and for variations among the sensorphotodetectors is also typically performed within the camera. Otherprocessing typically includes one or more of gamma correction, contraststretching, chrominance filtering and the like.

Electronic cameras also nearly always include an automatic exposurecontrol capability that sets the exposure time, size of its apertureopening and analog electronic gain of the sensor to result in theluminescence of the image or succession of images being at a certainlevel based upon calibrations for the sensor being used and userpreferences. These exposure parameters are calculated in advance of thepicture being taken, and then used to control the camera duringacquisition of the image data. For a scene with a particular level ofillumination, a decrease in the exposure time is made up by increasingthe size of the aperture or the gain of the sensor, or both, in order toobtain the data within a certain luminescence range. An increasedaperture results in an image with a reduced depth of field and increasedoptical blur, and increasing the gain causes the noise within the imageto increase. Conversely, when the scene is brightly lighted, theaperture and/or gain are reduced and compensated for by increasing theexposure time, the resulting image having a greater depth of fieldand/or reduced noise. In addition to analog gain being adjusted, or inplace of it, the digital gain of an image is often adjusted after thedata have been captured.

Other processing that may also be performed by electronic camerasincludes a detection of the likelihood that a certain type of object ispresent within the image. An example object is a human face. When thereis a likelihood that the object is present in the image, its location isalso determined. This allows the camera to act differently upon thatportion of the image during acquisition and/or processing of theacquired data.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present invention aredescribed with reference to the following drawings, in which:

FIG. 1 illustrates a block diagram of an embodiment of an imagingdevice;

FIG. 2 shows a simplified block diagram of an embodiment of a smilingface detection system; and

FIG. 3 illustrates a flowchart of an embodiment of a process of smilingface detection, in accordance with aspects of the invention.

DETAILED DESCRIPTION

Various embodiments of the present invention will be described in detailwith reference to the drawings, where like reference numerals representlike parts and assemblies throughout the several views. Reference tovarious embodiments does not limit the scope of the invention, which islimited only by the scope of the claims attached hereto. Additionally,any examples set forth in this specification are not intended to belimiting and merely set forth some of the many possible embodiments forthe claimed invention.

Throughout the specification and claims, the following terms take atleast the meanings explicitly associated herein, unless the contextdictates otherwise. The meanings identified below do not necessarilylimit the terms, but merely provide illustrative examples for the terms.The meaning of “a,” “an,” and “the” includes plural reference, and themeaning of “in” includes “in” and “on.” The phrase “in one embodiment,”as used herein does not necessarily refer to the same embodiment,although it may. Similarly, the phrase “in some embodiments,” as usedherein, when used multiple times, does not necessarily refer to the sameembodiments, although it may. As used herein, the term “or” is aninclusive “or” operator, and is equivalent to the term “and/or,” unlessthe context clearly dictates otherwise. The term “based, in part, on”,“based, at least in part, on”, or “based on” is not exclusive and allowsfor being based on additional factors not described, unless the contextclearly dictates otherwise. The term “signal” means at least onecurrent, voltage, charge, temperature, data, or other signal.

Briefly stated, the invention is related to a method, apparatus, andmanufacture for smiling face detection. For each frame, a list of newsmiling faces for the frame is generated by performing smiling facedetection employing an object classifier that is trained to distinguishbetween smiling faces and all objects in the frame that are not smilingfaces. For the first frame, the list of new smiling faces is employed asan input smiling face list for the next frame. For each frame after thefirst frame, a list of tracked smiles for the frame is generated bytracking smiling faces in the frame from the input smiling list for theframe. Further, a list of new smiling faces is generated for the nextframe by combining the list of new smiling faces for the frame with thelist of tracked smiles for the frame.

FIG. 1 shows a block diagram of an embodiment of device 100, which maybe a digital camera or the like. Digital camera 100 includes a set ofoptics (e.g., one or more lenses and/or light guides) 101, a set ofimage sensors 102 optically coupled to the optics 101, a set ofanalog-to-digital (A/D) converters 103 having inputs electricallycoupled to outputs of the image sensors 102, and one or more processorsand hardware 104 coupled to receive the outputs of the A/D converters103. The image sensors 102 may produce separate R, G and B colorsignals. Camera 100 further includes a display device 106 coupled tooutputs of the processor(s) and hardware 104, and a memory 105 havingbi-directional communication with the processor(s) 104. Display device106 is optional, and is not included in all embodiments of digitalcamera 100.

In operation, the image sensors 102 receive input light through theoptics 101 and, in response, produce analog output color signals R, Gand B to the A/D converters. The A/D converters convert those inputcolor signals to digital form, which are provided to the processor(s)104.

The processor(s) and hardware 104 may include a CPU as well asspecialized hardware, as discussed in greater detail below. Processor(s)104 may perform any of various well-known types of processing on thoseinput color signals. The processor(s) 104 may be or include, forexample, any one or more of: a programmed microprocessor or digitalsignal processor (DSP), a microcontroller, an application specificintegrated circuit (ASIC), a programmable logic device (PLD), etc.Processor(s) and hardware 104 may perform various processes, such aspart or all of an embodiment of the process illustrated in FIG. 3.

The memory 105 may be or include, for example, anyone or more of: flashmemory, read-only memory, random access memory (RAM), etc. Memory 105may include a tangible, processor-readable storage medium that arrangedto encode processor-readable code, which, when executed processor(s)104, enables actions. Actions enabled by processor(s) 104, which mayinclude action(s) controlled by processor(s) 104 but actually performedby other parts of digital camera 100, may perform various processes suchas part of all of an embodiment of the process illustrated in FIG. 3.

Digital camera 100 is not limited to consumer digital cameras, but mayinclude other types of imaging devices that capture images in a varietyof different manners.

Processed or raw color data can be output to the display device 106 fordisplay and/or to one or more external devices, such as a computer orprinter.

FIG. 2 depicts one embodiment of smile detection system 200, whichincludes image scatter unit 210, smiling face detection unit 230, andsmiling face tracking unit 220. Image scatter unit (ISU) 210 isconfigured to receive a data of a frame as input. Exemplary input to thesystem includes, but is not limited to, quarter video graphics array(QVGA), encoded image and/or movie data (e.g., YUV) and compressedimages (e.g., Y-only downscaled copies).

ISU 210 may be coupled to at least one of a detection device andprocessor for receiving data for the one or more frames. As used herein,a frame may relate to an image window or detection area of an imagingdevice, such as a photosensor, camera, video detector, etc. In certainembodiments, data received by ISU 210 may relate to a preview image ofthe detector. A frame may also describe detection data for a particularperiod of time. ISU 210 may be configured to output the received framedata for a device display. Following receipt of one or more frames, ISU210 is configured to provide the one or more frames to smiling facedetection unit (SD) 230 and smiling face tracking unit (ST) 220. SD 230and ST 220 are image processing components that are configured toperform actions. In one embodiment, ST 220 may employ a list of smilingface coordinates and scale for tracking one or more smiling faces.

SD 230 is configured to provide potential smiling face windows. Imagecapture and processing may be based on detection and focusing associatedwith the one or more windows within a frame. Based on one or morewindows supported by the image detection device, smiling face detectionmay be performed within the frame. SD 230 includes an object detectionunit that distinguishes between smiling faces and all objects that arenot smiling faces. In some embodiments, the smiling faces detected by SD230 may be combined with smiles tracked by ST 220 to provide a list ofsmiling faces to be tracked by ST 220 for the next frame, as discussedin greater detail below.

According to some embodiments of the invention, smile detection system200 may employ a list or set of one or more smiling faces to be trackedwithin a frame. ST 220 may be configured to provide a smiling face listthat may be used by an imaging device, such as a camera, including oneor more confidence levels associated with a detected smiling face. Theface list may be based on the degree of confidence, as being a realhuman smiling face, for each face in the face list. An output smilingface list 240 may be generated by filtering out tracked smiles which arebelow a particular threshold confidence level. In some embodiments, theconfidence level is a confidence level for each smiling face that istracked and averaged over time. For example, in some embodiments, outputsmiling face list 240 is generated by removing from the list of trackedsmiles each smiling face having an average confidence level that isbelow a threshold confidence level.

Although described as units of hardware in FIG. 2, it should beappreciated that the functions of the units may be implemented in avariety of ways including hardware, firmware, software, and/orcombinations thereof. In particular, in some embodiments, smiling facetracking unit 220 may be implemented as a combination of hardware andsoftware, and smiling face detection unit 230 may be implemented as acombination of hardware and software. The software components mayincludes a tangible, processor-readable storage medium that isconfigured to encode processor-readable code, which, when executed byone or more processors, enables actions.

FIG. 3 illustrates a flowchart of an embodiment of a process (350) ofsmiling face detection.

In some embodiments, process 350 may process a number of input images orframes. In some embodiments, the frames as associated with a previewimage of a digital camera. For each frame, a list of new smiling facesfor the frame is generated by performing smiling face detection atsmiling face detection unit 360, employing an object classifier that istrained to distinguish between smiling faces and all objects in theframe that are not smiling faces. For the first frame/input image, thelist of new smiling faces is employed as an input smiling face list forthe next frame/input image (i.e., the second frame/input image). Foreach frame after the first frame, a list of tracked smiling faces forthe frame is generated at smiling face tracking block 370 by trackingsmiling faces in the frame from the input smiling list for the frame.Further, a list of new smiling faces is generated for the next frame atcombining block 380 by combining the list of new smiling faces for theframe with the list of tracked smiles for the frame. In someembodiments, the list of tracked smiles may be filtered by clutterfilter 390 to generate an output smiling list.

In some embodiments, process 350 begins when a digital camera is set inauto-capture mode or smile detection mode to automatically capture astill image whenever a person is smiling in the camera field of view,with the camera responding quickly enough before the person stopssmiling. In some embodiments, a user may cause the digital camera toenter into the smile detection mode by selecting smile detection mode asan option from a menu. In some embodiments, the smile detection may beincluded within a self-timer mode, which takes a shot after a timer endswhen a smile is detected.

In some embodiments, smiling detection process 350 is preformed in acamera preview mode where the algorithm inputs are a sequence of images.The output of the algorithm is list of smiling faces that contains facescoordinates and size for each input image. In some embodiments, thesequence of input images may be images in the preview mode of thecamera. In various embodiments, the input frames may relate to a previewof image data, including still images, motion images, video, and/orimaged data.

Smiling face detection unit 360 is configured to find new faces over theentire input frame. In some embodiments, smiling face detection unit 360may accomplished using the objection detection unit described in U.S.Pat. No. 7,961,908 titled “Detecting objects in an image being acquiredby a digital camera or other electronic image acquisition device”, whichis hereby incorporated by reference. The object detection unit istrained with two prepared sets of databases. The first database includesa large number of smiling faces images of different persons. The set isused as a positive examples set. The second database includes a largenumber of images of different types of objects not including smilingfaces. The set is used as a negative examples set. Cascades ofclassifiers are prepared by training the classifiers with the positiveand negative sets. Then the object detection unit is fed with with thecascades of classifiers. In this way, the object detection unit isoperating as classifiers which detect the location and size of smilingfaces in the input image while rejecting any other type of objects.Accordingly, the object classifier is trained to reject all objects inthe image which are not smiling faces. Detection of smiling faces isaccomplished using the object-detection unit solely. The objectdetection unit distinguishes between smiling faces and all objects thatare not smiling faces. The training is performed offline, in factorybefore the parameters are placed in the camera. After the training isperformed, the parameters resulting from the training are loaded intothe camera.

In some embodiments, smiling face detection unit 360 operates as theobject detection unit described in U.S. Pat. No. 7,961,908, except thatrather than using the human face as the object being detected, thesmiling human face is the object being detected. The classifiersdistinguish between smiling human faces and all objects that are notsmiling human faces. Rather than being a two-stage process in whichfirst an object detection unit distinguishes between faces andnon-faces, and then a subsequent determination is made as to whether ornot the faces are smiling, smiling face detection unit 360 performs aone-stage process in which the object detection unit distinguishesbetween smiling faces and those objects that are not smiling faces.Rather than having the classifiers train between examples that are facesand examples that are non-faces, the classifiers train between examplesthat are smiling faces and examples that are not smiling faces.

Although an embodiment of object detection is described in greaterdetail in U.S. Pat. No. 7,961,908, briefly each classifier operates asfollows in one embodiment. For each frame, the classifier is configuredto establish boundaries of windows in the frame. The classifier thenevaluates data within individual windows with respect to stored data ofa first set of features of the particular type of object and assignsfirst scores to the individual windows that represent a likelihood ofthe presence of the first set of features of the particular type ofobject in the corresponding individual windows. The classifier thencompares the first scores with a predetermined first threshold todetermine a first group of windows having first scores indicative of thelikelihood of the presence of the first set of features of theparticular type of object and accordingly to reject those of theindividual windows other than those of the first group.

The first group of the windows is one or more but less than all of thewindows. The classifier then evaluates data within the individualselected windows of the first group, but not the rejected windows, withrespect to stored data of a second set of features of the particulartype of object. The classifier then assigns second scores to theindividual windows of the first group that represent the likelihood ofthe presence of the second set of features of the particular type ofobject in the corresponding individual windows of the first group. Theclassifier then compares the second scores with a predetermined secondthreshold to determine a second group of windows having second scoresindicative of the likelihood of the presence of the second set offeatures of the particular type of object, accordingly rejecting thoseof the individual windows of the first group other than those of thesecond group. As discussed above, the classifier is trained such thatthe “particular type of image” is a smiling face.

The classifiers operate in series such that the determination as towhether or not the windows includes a smiling face is more detailed andcomplicated each subsequent round. The windows not rejected by the firstclassifier to proceed to the next classifier, and so forth. Eachsubsequent classifiers is stronger than the previous classifiers in theseries such that the object must resemble a smiling face more and morein order to not be rejected in subsequent rounds. This allows windowsthat obviously do not contain smiling faces, such a window that containsnothing but a white wall in the background, to be rejected quickly.

In some embodiments, a different suitable method of object tracking maybe used in place of the method described in U.S. Pat. No. 7,961,908, butregardless the method should use object detection that distinguishesbetween smiling faces and those objects that are not smiling faces.

Smiling face tracking unit 370 is configured to track smiling faces inthe input image. In some embodiments, smiling unit 370 is implemented inthe same manner as the face tracking unit described in patentapplication 2010/0021008, titled “System and Method for Face Tracking”,herein incorporated by reference, except that smiling face tracking unit370 tracks smiling faces rather than faces. During training of smilingface tracking unit 370, smiling face tracking unit 370 is fed with listof smiling face objects instead of a tracking list of face objects asdescribed in U.S. patent application 2010/0021008.

In some embodiments, a different suitable method of tracking may be usedby smiling face tracking unit 370 in place of the method described inU.S. patent application 2010/0021008, but smiling faces are tracked byunit 370 rather than faces.

Combining unit 380 is configured to combine the list of new detectedsmiling faces and the tracked smiling faces to providing the list ofsmiling faces to be tracked by tracking 370.

It is possible for system 350 to wrongly detect clutter objects ornon-smiling faces as smiling faces. It is preferable to avoid falsedetection of smiling faces in the smile detection algorithm since thecamera could otherwise capture empty images where no face in the scene,or with faces that are not smiling. Clutter filter unit 390 isconfigured for reducing the number of false detections. Clutter filterunit 390 is configured to calculate the average detection confidencelevel over n frames (n>1) and pass only faces with an average detectionconfidence level that is higher than a predetermined threshold. Theconfidence level here relates to the confidence that the tracked smilingface is in fact a smiling face.

In some applications of some embodiments of process 350, the camera isfocusing on the smiling faces and capturing the image. But in someembodiments in some applications, the camera focusing time is criticaland the focusing process is done prior to smile detection. Accordingly,in some embodiments, the camera focuses on non-smiling faces in theimage and captures the image immediately when a non-smiling face startssmiling. In some embodiments in which camera focusing time is critical,this is accomplished by independently and concurrently executingface-detection and smiling-face detection applications and employing theface detection application for focusing and employing the smiling facedetection algorithm for image capturing.

Although various embodiments discussed above involve smiling facedetection and tracking, some embodiments may employ smiling facedetection only and not smiling face tracking, using the smiling facedetection method discussed above. For example, smiling face detectionmay be employed after capturing photo in still mode. These embodimentsand others are also within the scope and spirit of the invention.

The above specification, examples and data provide a description of themanufacture and use of the composition of the invention. Since manyembodiments of the invention can be made without departing from thespirit and scope of the invention, the invention also resides in theclaims hereinafter appended.

1-20. (canceled)
 21. A system for smiling face detection, comprising:means for generating a first list of new smiling faces for a first frameof a plurality of frames via distinguishing between smiling faces andobjects in the first frame that are not smiling faces; means forutilizing the first list of new smiling faces from the first frame as afirst input smiling face list for a second frame of the plurality offrames; means for tracking smiling faces in the second frame based onthe first input smiling face list for the second frame; and means forgenerating a second list of new smiling faces for the second frame viautilizing an object classifier trained to distinguish between smilingfaces and objects in the second frame that are not smiling faces. 22.The system of claim 21, further comprising: means for generating a listof tracked smiles for the second frame based on the tracking of thesmiling faces in the second frame; and means for generating a combinedlist of smiling faces as a second input smiling face list for a thirdframe of the plurality of frames via combining the second list of newsmiling faces for the second frame with the list of tracked smiles forthe second frame.
 23. The system of claim 21, wherein the plurality offrames are associated with a preview image of a digital camera.
 24. Thesystem of claim 21, wherein the tracking means further comprises meansfor detecting an image in the second frame based on at least one ofcoordinate or scale values of smiling faces in the first input smilinglist for the second frame.
 25. The system of claim 21, furthercomprising means for generating a list of output smiling faces viafiltering the second list of new smiling faces for the second frame. 26.The system of claim 25, wherein filtering the second list of new smilingfaces for the second frame comprises removing from the second list ofnew smiling faces for the second frame each smiling face having anaverage confidence level that is below a threshold confidence level. 27.The system of claim 26, wherein the threshold confidence level comprisesa confidence level for each smiling face that is tracked and averagedover time.
 28. A method for smiling face detection, comprising:generating a first list of new smiling faces for a first frame of aplurality of frames via distinguishing between smiling faces and objectsin the first frame that are not smiling faces; utilizing the first listof new smiling faces from the first frame as a first input smiling facelist for a second frame of the plurality of frames; tracking smilingfaces in the second frame based on the first input smiling face list forthe second frame; and generating a second list of new smiling faces forthe second frame via utilizing an object classifier trained todistinguish between smiling faces and objects in the second frame thatare not smiling faces.
 29. The method of claim 28, further comprising:generating a list of tracked smiles for the second frame based on thetracking of the smiling faces in the second frame; and generating acombined list of smiling faces as a second input smiling face list for athird frame of the plurality of frames via combining the second list ofnew smiling faces for the second frame with the list of tracked smilesfor the second frame.
 30. The method of claim 28, wherein the pluralityof frames are associated with a preview image of a digital camera. 31.The method of claim 28, wherein tracking smiling faces in the secondframe comprises detecting an image in the second frame based on at leastone of coordinate or scale values of smiling faces in the first inputsmiling list for the second frame.
 32. An apparatus for smiling facedetection, comprising: a plurality of image sensors configured toreceive input light; and an image processor coupled to the plurality ofimage sensors configured to: generate a first list of new smiling facesfor a first frame of a plurality of frames via distinguishing betweensmiling faces and objects in the first frame that are not smiling facesutilize the first list of new smiling faces from the first frame as afirst input smiling face list for a second frame of the plurality offrames; track smiling faces in the second frame based on the first inputsmiling face list for the second frame; and generate a second list ofnew smiling faces for the second frame via utilizing an objectclassifier trained to distinguish between smiling faces and objects inthe second frame that are not smiling faces.
 33. The apparatus of claim32, wherein the image processor is further configured to: generate alist of tracked smiles for the second frame based on the tracked smilingfaces in the second frame; and generate a combined list of smiling facesas a second input smiling face list for a third frame of the pluralityof frames via combining the second list of new smiling faces for thesecond frame with the list of tracked smiles for the second frame. 34.The apparatus of claim 32, wherein the plurality of frames areassociated with a preview image of a digital camera.
 35. The apparatusof claim 32, wherein the image processor is further configured to detectan image in the second frame based on at least one of coordinate orscale values of smiling faces in the first input smiling list for thesecond frame.
 36. A non-transitory computer readable storage mediumhaving stored thereon instructions that, when executed, cause aprocessor of a device to: generate a first list of new smiling faces fora first frame of a plurality of frames via distinguishing betweensmiling faces and objects in the first frame that are not smiling faces;utilize the first list of new smiling faces from the first frame as afirst input smiling face list for a second frame of the plurality offrames; track smiling faces in the second frame based on the first inputsmiling face list for the second frame; and generate a second list ofnew smiling faces for the second frame via utilizing an objectclassifier trained to distinguish between smiling faces and objects inthe second frame that are not smiling faces.
 37. The non-transitorycomputer readable storage medium of claim 36, further having storedthereon instructions that, when executed, cause the processor to:generate a list of tracked smiles for the second frame based on thetracked smiling faces in the second frame; and generate a combined listof smiling faces as a second input smiling face list for a third frameof the plurality of frames via combining the second list of new smilingfaces for the second frame with the list of tracked smiles for thesecond frame.
 38. The non-transitory computer readable storage medium ofclaim 36, further having stored thereon instructions that, whenexecuted, cause the processor to detect an image in the second framebased on at least one of coordinate or scale values of smiling faces inthe first input smiling list for the second frame.
 39. Thenon-transitory computer readable storage medium of claim 36, furtherhaving stored thereon instructions that, when executed, cause theprocessor to generate a list of output smiling faces via filtering thesecond list of new smiling faces for the second frame.
 40. Thenon-transitory computer readable storage medium of claim 36, furtherhaving stored thereon instructions that, when executed, cause theprocessor to remove from the second list of new smiling faces for thesecond frame each smiling face having an average confidence level thatis below a threshold confidence level.